Apparatus and method for four dimensional soft tissue navigation

ABSTRACT

A surgical instrument navigation system is provided that visually simulates a virtual volumetric scene of a body cavity of a patient from a point of view of a surgical instrument residing in the cavity of the patient. The surgical instrument navigation system includes: a surgical instrument; an imaging device which is operable to capture scan data representative of an internal region of interest within a given patient; a tracking subsystem that employs electro-magnetic sensing to capture in real-time position data indicative of the position of the surgical instrument; a data processor which is operable to render a volumetric, perspective image of the internal region of interest from a point of view of the surgical instrument; and a display which is operable to display the volumetric perspective image of the patient.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.Nos. 61/375,439, filed Aug. 20, 2010, 61/375,484, filed Aug. 20, 2010,61/375,523, filed Aug. 20, 2010, and 61/375,533, filed Aug. 20, 2010,each of which are hereby incorporated by reference in their entirety,including any figures, tables, and drawings.

BACKGROUND

The invention relates generally to a medical device and particularly toan apparatus and methods associated with a range of image guided medicalprocedures.

Image guided surgery (IGS), also known as image guided intervention(IGI), enhances a physician's ability to locate instruments withinanatomy during a medical procedure. IGS can include 2-dimensional (2-D),3-dimensional (3-D), and 4-dimensional (4-D) applications. The fourthdimension of IGS can include multiple parameters either individually ortogether such as time, motion, electrical signals, pressure, airflow,blood flow, respiration, heartbeat, and other patient measuredparameters.

Existing imaging modalities can capture the movement of dynamic anatomy.Such modalities include electrocardiogram (ECG)-gated orrespiratory-gated magnetic resonance imaging (MRI) devices, ECG-gated orrespiratory-gated computer tomography (CT) devices, standard computedtomography (CT), 3D Fluoroscopic images (Angio-suites), andcinematography (CINE) fluoroscopy and ultrasound. Multiple imagedatasets can be acquired at different times, cycles of patient signals,or physical states of the patient. The dynamic imaging modalities cancapture the movement of anatomy over a periodic cycle of that movementby sampling the anatomy at several instants during its characteristicmovement and then creating a set of image frames or volumes.

A need exists for an apparatus that can be used with such imagingdevices to capture pre-procedural or intra-procedural images of atargeted anatomical body and use those images intra-procedurally to helpguide a physician to the correct location of the anatomical body duringa medical procedure.

SUMMARY OF THE INVENTION

A method includes receiving during a first time interval image dataassociated with an image of a dynamic body. The image data includes anindication of a position of a first marker on a patient tracking device(PTD) coupled to the dynamic body and a position of a second marker onthe PTD. Some registration methods such as 2D to 3D registrationtechniques allow for the image data containing the target or patientanatomy of interest to not contain the PTD. A registration step isperformed to calculate the transformation from image space to patientspace using an additional dataset to register (i.e., a 2D fluoroscopicset of images is used to register a 3D fluoroscopic dataset). Thistechnique is not limited to fluoroscopic procedures as it canimplemented in any procedure acquiring 2D images such as ultrasound, OCT(optical coherence tomography), EBUS (endobronchial ultrasound), or IVUS(intravascular ultrasound). This technique uses the markers that arewithin multiple 2D images to register the 3D volume that isreconstructed from these 2D images. The reconstructed 3D volume issmaller than the field of view of the 2D images, so this techniqueallows for the PTD markers to be visible in a subset of the 2D images,but not within the 3D volume. In certain embodiments, the first markeris coupled to the PTD at a first location and the second marker iscoupled to the PTD at a second location. A distance between the positionof the first marker and the position of the second marker is determined.During a second time interval after the first time interval, dataassociated with a position of a first localization element coupled tothe PTD at the first location and data associated with a position of asecond localization element coupled to the PTD at the second locationare received. A distance between the first localization element and thesecond localization element based on the data associated with theposition of the first localization element and the position of thesecond localization element is determined. A difference is calculatedbetween the distance between the first marker and the second markerduring the first time interval and the distance between the firstlocalization element and the second localization element during thesecond time interval. In addition the PTD device can be trackedcontinuously during the procedure and a sequence of motion of the PTDdevice that represents the patient motion of an organ or the patient'srespiratory cycle can be collected. The sequence of motion can then beanalyzed to find unique similar points within the dataset and grouped.

Other objects and features will be in part apparent and in part pointedout hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of the present invention, both as to its construction andoperation can best be understood with reference to the accompanyingdrawings, in which like numerals refer to like parts, and in which:

FIG. 1 is a schematic illustration of various devices used with a methodaccording to an embodiment of the invention.

FIG. 2 is a schematic illustration of various devices used with a methodaccording to an embodiment of the invention.

FIG. 3 is a schematic illustrating vector distances on an apparatusaccording to an embodiment of the invention.

FIG. 4A is a schematic illustrating vector distances from a localizationdevice according to an embodiment of the invention.

FIG. 4B is a schematic illustrating vector distances from image dataaccording to an embodiment of the invention.

FIG. 5 is a front perspective view of an apparatus according to anembodiment of the invention.

FIG. 6 is a graphical representation illustrating the function of anapparatus according to an embodiment of the invention.

FIG. 7 is a flowchart illustrating a method according to an embodimentof the invention.

FIG. 8 shows the layout of a system that may be used to carry out imageguided interventions using certain of the present methods that involvegated datasets.

FIG. 9 illustrates one example of samples of a periodic humancharacteristic signal (specifically, an ECG waveform) associated, orgated, with images of dynamic anatomy.

FIG. 10 is a diagram of an exemplary surgical instrument navigationsystem in accordance with present invention;

FIG. 11 is a flowchart that depicts a technique for simulating a virtualvolumetric scene of a body cavity from a point of view of a surgicalinstrument positioned within the patient in accordance with the presentinvention;

FIG. 12 is an exemplary display from the surgical instrument navigationsystem of the present invention;

FIG. 13 is a flowchart that depicts a technique for synchronizing thedisplay of an indicia or graphical representation of the surgicalinstrument with cardiac or respiratory cycle of the patient inaccordance with the present invention; and

FIG. 14 is a flowchart that depicts a technique for generatingfour-dimensional image data that is synchronized with the patient inaccordance with the present invention.

FIG. 15 is a graph depicting an axis or point that the instrument (e.g.,a bronchoscope) deflects in a single planar direction. The graph showsthe instrument (e.g., a bronchoscope) being maneuvered in six differentorientations in a 3D localizer volume, with all orientations convergingabout a common axis or point of deflection.

FIG. 16 is a graph depicting the eigenvalues (e0,e1,e2) for a moving 3.0sec PCA (principal component analysis) window over a data file including1800 samples. The square wave represents an on/off “wiggle detector”state based on the algorithm described herein, and this square wavedemonstrates that the algorithm exhibits no false negatives for thevalidation test data and that the seven exemplary “wiggle” periods areclearly matched to the “on” state of the wiggle detector. Theimplementation of the algorithm uses low pass filtering and anappropriate comparator function to eliminate any false positive tracesor spots (“blips”) indicated in FIG. 16.

FIG. 17 is an image of an exemplary synthetic radiograph in accordancewith the present invention, depicting the historical instrument positiontrace.

FIG. 18 depicts an exemplary curvature warning system in accordance withthe invention described herein.

FIG. 19 depicts an exemplary real-time respiration compensationalgorithm.

FIG. 20 depicts additional and alternative exemplary embodiments of theinvention described herein.

FIG. 21 depicts additional and alternative exemplary embodiments of theinvention described herein.

FIGS. 22A and 22B depict additional and alternative exemplaryembodiments of the invention described herein.

FIG. 23 depicts additional and alternative exemplary embodiments of theinvention described herein.

FIGS. 24A and 24B depict exemplary embodiments of an port offset devicein accordance with the invention described herein.

FIG. 25 depicts exemplary embodiments of a actuatable sensor-equippedforceps device in accordance with the invention described herein.

DETAILED DESCRIPTION

The accompanying Figures and this description depict and describeembodiments of a navigation system (and related methods and devices) inaccordance with the present invention, and features and componentsthereof. It should also be noted that any references herein to front andback, right and left, top and bottom and upper and lower are intendedfor convenience of description, not to limit the present invention orits components to any one positional or spatial orientation.

It is noted that the terms “comprise” (and any form of comprise, such as“comprises” and “comprising”), “have” (and any form of have, such as“has” and “having”), “contain” (and any form of contain, such as“contains” and “containing”), and “include” (and any form of include,such as “includes” and “including”) are open-ended linking verbs. Thus,a method, an apparatus, or a system that “comprises,” “has,” “contains,”or “includes” one or more items possesses at least those one or moreitems, but is not limited to possessing only those one or more items.For example, a method that comprises receiving a position of aninstrument reference marker coupled to an instrument; transforming theposition into image space using a position of a non-tissue internalreference marker implanted in a patient; and superimposing arepresentation of the instrument on an image in which the non-tissueinternal reference marker appears possesses at least the receiving,transforming, and superimposing steps, but is not limited to possessingonly those steps. Accordingly, the method also covers instances wherethe transforming includes transforming the position into image spaceusing a transformation that is based, in part, on the position of thenon-tissue internal reference marker implanted in the patient, andcalculating the transformation using image space coordinates of theinternal reference marker in the image. The term “use” should beinterpreted the same way. Thus, a calculation that uses certain itemsuses at least those items, but also covers the use of additional items.

Individual elements or steps of the present methods, apparatuses, andsystems are to be treated in the same manner. Thus, a step that callsfor creating a dataset that includes images, one of the images (a)depicting a non-tissue internal reference marker, (b) being linked tonon-tissue internal reference marker positional information, and (c)being at least 2-dimensional covers the creation of at least such adataset, but also covers the creation of a dataset that includes images,where each image (a) depicts the non-tissue internal reference marker,and (b) is linked to non-tissue internal reference marker positionalinformation.

The terms “a” and “an” are defined as one or more than one. The term“another” is defined as at least a second or more. The term “coupled”encompasses both direct and indirect connections, and is not limited tomechanical connections.

Those of skill in the art will appreciate that in the detaileddescription below, certain well known components and assembly techniqueshave been omitted so that the present methods, apparatuses, and systemsare not obscured in unnecessary detail.

An apparatus according to an embodiment of the invention includes a PTDand two or more markers coupled to the PTD. The apparatus can alsoinclude two or more localization elements coupled to the PTD proximatethe markers. The apparatus is configured to be coupled to a dynamicbody, such as selected dynamic anatomy of a patient. Dynamic anatomy canbe, for example, any anatomy that moves during its normal function(e.g., the heart, lungs, kidneys, liver and blood vessels). A processor,such as a computer, is configured to receive image data associated withthe dynamic body taken during a pre-surgical or pre-procedural firsttime interval. The image data can include an indication of a position ofeach of the markers for multiple instants in time during the first timeinterval. The processor can also receive position data associated withthe localization elements during a second time interval in which asurgical procedure or other medical procedure is being performed. Theprocessor can use the position data received from the localizationelements to determine a distance between the elements for a giveninstant in time during the second time interval. The processor can alsouse the image data to determine the distance between the markers for agiven instant in time during the first time interval. The processor canthen find a match between an image where the distance between themarkers at a given instant in time during the first time interval is thesame as the distance between the elements associated with those markersat a given instant in time during the medical procedure, or second timeinterval. Additionally, the processor can determine a sequence of motionof the markers and match this sequence of motion to the recorded motionof the markers over the complete procedure or significant period oftime. Distance alone between the markers may not be sufficient to matchthe patient space to image space in many instances, it is important forthe system to know the direction the markers are moving and the rangeand speed of this motion to find the appropriate sequence of motion fora complex signal or sequence of motion by the patient.

A physician or other healthcare professional can use the images selectedby the processor during a medical procedure performed during the secondtime interval. For example, when a medical procedure is performed on atargeted anatomy of a patient, such as a heart or lung, the physicianmay not be able to utilize an imaging device during the medicalprocedure to guide him to the targeted area within the patient. A PTDaccording to an embodiment of the invention can be positioned or coupledto the patient proximate the targeted anatomy prior to the medicalprocedure, and pre-procedural images can be taken of the targeted areaduring a first time interval. Markers or fiducials coupled to the PTDcan be viewed with the image data, which can include an indication ofthe position of the markers during a given path of motion of thetargeted anatomy (e.g., the heart) during the first time interval. Suchmotion can be due, for example, to inspiration (i.e., inhaling) andexpiration (i.e., exhaling) of the patient, or due to the heart beating.During a medical procedure, performed during a second time interval,such as a procedure on a heart or lung, the processor receives data fromthe localization elements associated with a position of the elements ata given instant in time during the medical procedure (or second timeinterval). The distance between selected pairs of markers can bedetermined from the image data and the distance, range, acceleration,and speed between corresponding selected pairs of localization elementscan be determined based on the element data for given instants in time.From multiple image datasets the range and speed of the markers motioncan be calculated.

Because the localization elements are coupled to the PTD proximate thelocation of the markers, the distance between a selected pair ofelements can be used to determine an intra-procedural distance betweenthe pair of corresponding markers to which the localization elements arecoupled. An image from the pre-procedural image data taken during thefirst time interval can then be selected where the distance between thepair of selected markers in that image corresponds with or closelyapproximates the same distance determined using the localizationelements at a given instant in time during the second time interval.This process can be done continuously during the medical procedure,producing simulated real-time, intra-procedural images illustrating theorientation and shape of the targeted anatomy as a catheter, sheath,needle, forceps, guidewire, fiducial delivery devices, therapy device(ablation modeling, drug diffusion modeling, etc.), or similarstructure(s) is/are navigated to the targeted anatomy. Thus, during themedical procedure, the physician can view selected image(s) of thetargeted anatomy that correspond to and simulate real-time movement ofthe anatomy. In addition, during a medical procedure being performedduring the second time interval, such as navigating a catheter or otherinstrument or component thereof to a targeted anatomy, the location(s)of a sensor (e.g., an electromagnetic coil sensor) coupled to thecatheter during the second time interval can be superimposed on an imageof a catheter. The superimposed image(s) of the catheter can then besuperimposed on the selected image(s) from the first time interval,providing simulated real-time images of the catheter location relativeto the targeted anatomy. This process and other related methods aredescribed in pending U.S. patent application Ser. No. 10/273,598,entitled Methods, Apparatuses, and Systems Useful in Conducting ImageGuided Interventions, filed Nov. 8, 2003, the entire disclosure of whichis incorporated herein by reference.

In one embodiment, a real-time pathway registration is applied to apre-acquired dataset that does not contain the PTD. It will beunderstood that the pre-acquired dataset can be at only one cycle of apatient's respiratory, heartbeat, or other path of motion. In order tooptimize the registration of a pre-acquired dataset that does notcontain the PTD, a PTD can be subsequently applied to the patient, andthe PTD signal can be used to collect registration informationthroughout full range or path of motion but only that information thatis captured at a similar PTD orientation, shape, or point along the PTDcycle of motion is used. This method enhances the registration accuracyby ensuring that the registration points being used to register are atthe same point during the initial dataset acquisition. In preferredembodiments, the method uses multiple subsets of the acquiredregistration data that are collected based on the PTD signal. Thesemultiple subsets are then applied against the pre-acquired dataset tofind the optimal registration fit.

In another embodiment, the device can be integrated with one or morefiber optic localization (FDL) devices and/or techniques. In this way,the sensor (such as an EM sensor) provides the 3D spatial orientation ofthe device, while the FDL provides shape sensing of the airway, vessel,pathway, organ, environment and surroundings. Conventional FDLtechniques can be employed. In various embodiments, for example, the FDLdevice can be used to create localization information for the completepathway or to refine the localization accuracy in a particular segmentof the pathway. By either using 3D localization information, shape, orboth detected by the FDL device, the system can use a weighted algorithmbetween multiple localization devices to determine the location andorientation of the instrument in the patient. The FDL device can also beused as or in conjunction with the PTD to track the patient's motionsuch as respiration or heartbeat.

Other aspects involve using a guidewire or other navigated instrumentwith one to one rotation to continuously align a virtual display view tobe consistent with the actual bronchoscopic video view. A similartechnique can be used with OCT, IVUS, or EBUS devices to orient thevirtual view to the image captured by the OCT, IVUS, or EBUS devices.

Other aspects involve using video input of the bronchoscope to adjustthe virtual “fly-through” view to be consistent with the user's normalperspective. For example, conventional video processing and matchingtechniques can be used to align the real-time video and the virtualimage.

Other aspects involve using bronchoscopic video to provide angularinformation at a current location to provide targeting or directionalcues to the user. Angular information can be derived from the locationof patient anatomy in the image and the relative size of each within theimage. Using information extracted from the video captured by thebronchoscope, the system can determine which the direction of thedisplay. This can be done using, for example, translation, rotation, ora combination of both. Comparing the real-time image captured to thevirtual image constructed from the 3D dataset (i.e., CT) the system canuse this information to align the virtual image and/or enhance thesystem accuracy.

In another aspect, a high-speed three-dimensional imaging device, suchas an optical coherence tomography (OCT) device, can be tracked. Inaccordance with conventional methods, such a device can only view 1-2 mmbelow the surface. With an EM sensor attached in accordance with thesystems and methods described herein, multiple 3D volumes of data can becollected and a larger 3D volume of collected data can be constructed.Knowing the 3D location and orientation of the multiple 3D volumes willallow the user to view a more robust image of, for example,pre-cancerous changes in the esophagus or colon. This data can also becorrelated to pre-acquired or intra-procedurally acquired CT,fluoroscopic, ultrasound, or 3D fluoroscopic images to provideadditional information.

Among several potential enhancements that could be provided by anendolumenal system as described herein is that a user could overlay theplanned pathway information on to the actual/real-time video image ofthe scope or imaging device (such as ultrasound based device).Additionally, the system and apparatus could provide a visual cue on thereal-time video image showing the correct direction or pathway to take.

Additionally, or alternatively, one could use a 5DOF sensor and alimited or known range of motion of a localization device to determineits orientation in the field. This is particularly relevant, forexample, in determining which way is up or down or the overall rotationof an image in 3D space. In bronchoscopy, for instance, this can be usedto orient the bronchoscopic view to the user's normal expected visualorientation. Because a bronchoscope is typically only able to move inone plane up and down, the system can use the 3D location of a tipsensor moving up and down to determine the sixth degree of freedom. Inthis implementation, a user could steer the bronchoscope to abifurcation or close to a bifurcation and then perform a motion with thescope (e.g., up and down) using the thumb control to wiggle or flutterthe tip and sensor. With this motion (i.e., described herein generallyas the “wiggle maneuver”), the system can determine the orientation anddisplay the correct image. Typically, 5DOF sensing of instrument tipPOSE (position and orientation) determines 5 of the 6 POSE parameters(x, y, z, pitch, and yaw). Such sensing may be unable in certainapplications, however, to determine the instantaneous roll of thedevice. This roll determination can be critical in matching the videocoming from the device to the images. The techniques described hereinadvantageously allow for users to relate orientation of a virtualendoscopic “fly-thru” display to actual video orientation presented bythe endoscopic instrument.

In general, the methods described herein provides for the user to selecta location, most typically at a branching point in the bronchial tree,and perform a “wiggle” maneuver with the tip of the device. In preferredembodiments, the wiggle maneuver generally consists of three steps:

(i). At desired branch or other point, the physical or translationallocation of device is substantially secured or held in place. Forexample, with a bronchoscope, the user should ensure that the scope isheld securely so it cannot translate relative to the airway.

(ii). Perform a tip wiggle in plane by a rhythmic actuation of the scopesteering mechanism. The magnitude of the actuation should be sufficientin force for the tip to cover an approximate 1 cm-2 cm range of motion,but less motion may be sufficient in some applications or embodiments.

(iii). Continue the wiggle maneuver, keeping the scope itselfsubstantially stationary until the systems described herein and relatedsoftware defines the sixth degree of freedom, and the orientation of thevideo display of scope matches the virtual “fly-thru”.

Algorithmically, a manifestation of an algorithm to detect thisoperation consists of recognizing a unique signature of motion overtime. One such technique, for example, consists of two parts:

(a) performance of a continuous PCA analysis of a specified time windowof 5DOF sensor locations. A repeated motion in a plane by the instrumenttip will produce a covariance matrix such that the foremost eigenvalue(e0) will reflect the variance of a 1 cm-2 cm motion over a given timewindow. In addition, the secondary and tertiary eigenvalues (e1 and e2)will reflect a very small variance, as the tip should preferably bemoving in an arc constrained to the plane defined by the splinemechanism of the scope.

(b) once an appropriate PCA signature is detected, orientation of thetip to the eigenvector E0, which represents the historical vector ofmotion for the instrument tip, is compared. An exemplary way to do thisis the dot product of the measured tip vector and E0, which representsthe acute angle.

r=V·E0

By way of example and not by way of limitation, in an ideal wigglemaneuver, the orientation of the tip should show a rhythmic oscillationof about 90° (approximate range could be for instance +/−)45°. Thiscomparison of tip orientation to E0 provides the basis for adetermination of the wiggle plane normal using, for example, across-product or gram-schmidt orthogonalization. The physical wiggleplane normal is in a constant relationship to the coordinate space ofthe video signal, and thus can be related (calibrated) thereto.

Interestingly, this algorithm can be used in a variety of differentmodes to detect certain forms of scope motion:

-   -   stationary scope (wherein the e0, e1, and e2 values will be very        small and roughly equal, representing the small magnitude,        unbiased Gaussian noise of the direct localization measurement);    -   scope in motion along a straight line in space, such as when        traversing individual segments (wherein the e0 will be very        large, with very small e1 and e2 values, representing a large        co-linear translation in space). In addition, the absolute value        of r will be close to 1, indicating that the orientation of the        tip is nearly collinear with the translation of the tip over        time.

In general, the wiggle techniques described herein for determiningdirection and orientation, including “up” and “down” orientationrelative to (or independent of) the instrument steering mechanism, mayused in a range of different endoscopic applications. Although the aboveexamples and embodiments generally refer to the use of the wigglemaneuver in connection with bronchoscopic applications, it will beunderstood that the techniques described herein may also be used inother applications including, but not limited to, enteroscopy,colonoscopy, sigmoidoscopy, rhinoscopy, proctoscopy, otoscopy,cystoscopy, gynoscopy, colposcopy, hysteroscopy, falloposcopy,laparoscopy, arthroscopy, thoracoscopy, amnioscopy, fetoscopy,panendoscopy, epiduroscopy, and the like. The wiggle techniquesdescribed herein may also be applicable in non-medical endoscopic uses,such as the internal inspection of complex technical systems,surveillance, and the like.

In general, the methods described herein employ successive approximationtechniques of pathway planning, and enhanced localization accuracy touse traveled pathway information to continuously, substantiallycontinuously, or serially update pathway planning and localizationaccuracy. The limits of automatic pathway or vessel segments aregenerally defined, for example, by the quality of the image dataprovided or the state the patient may be in when the image data isacquired. Therefore, in some cases the pathway segmentation may notextend completely to the target (that is, the pathway segmentation stopsshort of the target). One method to enhance the segmentation involvesusing the traveled path of a tracked instrument to add branches andsegment(s) to the pathway. Methods may include multiple approachesdescribed herein such as recording traveled paths or using theadditional path traveled information to iteratively segment vessels orairways from the image dataset. Having the additional knowledge that aninstrument actually traveled the path can enable an automaticsegmentation algorithm to extend the vessels and airways that it cancalculate and find. This can be particularly valuable in situationswhere airways or vessels may be obstructed or pinched off. Thistechnique can also be a valuable tool in recording the paths traveled bythe instrument to give an indication to the user of “good” and “bad”(i.e., correct or incorrect, direct or indirect, etc.) directions based,for instance, on trial and error.

According to the various methods and systems described herein,therefore, in some embodiments real-time pathway registration uses full5D and/or 6D information and/or details of instrument location toenhance registration. These and other methods can employ distance,proximity, and/or angle information, for example, to determine thepathway being traversed in order to optimize localization accuracy. Thiscan be particularly beneficial at the bifurcation(s) of vessels andairways, where an initial (or subsequent) selection of the correctdirection is important to accurately navigate the desired pathway. Usingboth 3D localization and the orientational and/or directional angle ofthe device can, in various embodiments, be helpful in maximizingaccuracy in applications such as an endolumenal navigation system.Advance knowledge of the various segmented vessels and/or airways in aprocedural or pre-procedural setting, and the orientation and/ordirection of the navigated instrument, can further enhance accuracy. Incertain embodiments, the various parameters (e.g., distance, angle,etc.) can be weighted in real-time (or near-real-time) for improvedapplication or use of the relevant information. For example, distanceinformation can be used in the relevant algorithms without taking angleinformation into account, or vice versa. Or, for example, the variousparameters can be equally applied (e.g., 50% angle, 50% proximity).Additionally or alternatively, different percentages can be applied toemphasize or weigh one parameter over another (e.g., 15% angle, 85%proximity; 20% angle, 80% proximity; 25% angle, 75% proximity; 30%angle, 70% proximity; 35% angle, 65% proximity; 40% angle, 60%proximity; or 45% angle, 65% proximity; or, alternatively, 15%proximity, 85% angle; 20% proximity, 80% angle; 25% proximity, 75%angle; 30% proximity, 70% angle; 35% proximity, 65% angle; 40%proximity, 60% angle; 45% proximity, 65% angle). In operation, it may bepreferable to more heavily weigh angle information when the instrumentarrives at a branch (e.g., an arterial branch) in order to facilitate orimprove directional decision-making in real-time; weighting theparameters in this manner, therefore, can further improve accuracy.

According to one embodiment, the real-time pathway registration usespath traveled of the instrument location to enhance registration. Usingpath traveled to exclude other potential navigational solutions enhancesnavigational accuracy. Many times in patient anatomy vessel or airwaybranches may twist around and come close to one another, but it isobvious from earlier location(s) recorded along the path traveled thatonly one branch can be the correct navigated location.

According to one particular embodiment, the respiratory signal derivedfrom the PTD is used to gate the localization information of theinstrument in the airway. This can assist in determining multiple airwaymodels, for example, by performing a best fit of the real-time patientairway model to the CT data to derive the optimal registration and gatedperiod in the patient's respiratory cycle. Additionally oralternatively, the respiratory signal can be derived from devices otherthan the PTD, such a device that records the resistance between twolocations on the patient. For example, this method is similar to avariable potentiometer in that the resistance of the patient changesbetween two fixed points as the patient inhales and exhales. Thus, theresistance can be measured to create a respiratory signal.

According to another particular embodiment, 3D location information isused to extend the segmented airway model. The 3D airway can be extendedas the instrument is passed along the airway by using this locationinformation as an additional parameter to segment the airway from the CTdata. Using an iterative segmentation process, for instance, the 3Dlocation information of the instrument can be used to provide seedpoints, manual extension, or an additional variable of likelihood of asegmented vessel or airway existing in the 3D image volume. These addedairways can be displayed in a different format or color (for example) toindicate to the user that they are extending the segmented airway usinginstrument location information.

In general, the embodiments described herein have applicability in“Inspiration to Expiration”-type CT scan fusion. According to variousmethods, the user navigates on the expiration CT scan for optimalaccuracy, while using the inspiration scan to obtain maximum airwaysegmentation. In one embodiment, for example, a user could completeplanning and pathway segmentation on an inspiration scan of the patient.Preferably, a deformation or vector field is created between at leasttwo datasets. The deformation or vector field may then be applied to thesegmented vessels and/or airways and the user's planned path and target.In these and other embodiments, the deformation or vector field can alsobe applied to multiple datasets or in a progressive way to create amoving underlying dataset that matches the patient's respiratory orcardiac motion.

By way of example, “Inspiration to Expiration” CT fusion using the lunglobe centroid and vector change to modify an airway model may also beapplicable. In accordance with various embodiments, this technique maybe used to translate and scale each airway based on the lung lobe changebetween scans. The lung is constructed of multiple lobes and these lobesare commonly analyzed for volume, shape, and translation change. Eachlobe changes in a very different way during the patient's breathingcycle. Using this information to scale and translate the airways thatare located in each lobe, it is possible to adapt for airway movement.This scaled airway model can then be linked to the 4D tracking of thepatient as described herein.

In one preferred embodiment, for example, a cine loop of ultrasound datais collected in conjunction with the patient's respiratory cycleinformation. This can serve to limit registration point selection, inorder to be consistent with the patient's respiratory cycle that a 3Ddataset such as CT, MR, or PET has acquired. This techniqueadvantageously maximizes registration accuracy, a major flaw inconventional systems in the prior art.

In various aspects, the systems and methods described herein involvemodifying inspiration CT scans to the expiration cycle for navigation.It is well understood that the patient's airways are contained withinmultiple lobes of the lung. It is also understood that airwayssignificantly change between inspiration and expiration. In order tohave the most accurate map for navigation it would be beneficial toinclude the detail of the inspiration scan, coupled with the ability tonavigate it accurately during expiration, which is the most repeatablepoint in a patient's breath cycle. In preferred embodiments, thismodification can be carried out in accordance with the following steps:

1) Scan patient at both inspiration and expiration;

2) Segment the airways in both the inspiration and expiration;

3) Segment the lung lobes in both the inspiration and expiration (as thelung lobes are identifiable in both the inspiration and expiration scanswith a high degree of accuracy);

4) Determine a volume difference for each lung lobe between inspirationand expiration, use this change to shrink the airway size from theinspiration to the expiration cycle. Preferably, this is done for eachindividual lobe, as the percentage change will typically be differentfor each lobe.

5) Determine the centroid for each lung lobe and the vector change inmotion from the main carina in both scans. This vector can then be usedto shift the airways that are associated with each lung lobe. A centroidfor the airway can be calculated based on the segmented branches. Foreach airway branch in the segmentation, it includes a tag thatassociates it with the respective lung lobe. The central airwayincluding the main carina and initial airway branches for each lobe thatis linked according to the expiration scan location of these points.Next, a plane can be defined using the main carina and initial airwaybranch exits to determine the vector change for each lobe.

Among the lobes to modify, for example:

left inferior lobe—the bottom lobe of the lung on the left side of thebody.

left superior lobe—the top lobe of the lung on the left side of thebody.

right inferior lobe—the bottom lobe of the lung on the right side of thebody.

right middle lobe—the middle lobe of the lung on the right side of thebody.

right superior lobe—the top lobe of the lung on the right side of thebody.

Exemplary calculations are as follows:

Inspiration Airway−Left Inferior Lobe(LIL)×70%(reduction in volumeInspiration to Expiration calculated)=ExAirwayLlL

Determine Expiration Central Airway points (Main Carina and InitialAirway branches) based upon segmentation

Shift ExAirwayLIL by vector distance (3 cm, 45 degrees up and back frommain carina) that LIL centroid moved from inspiration to expiration

Preferably, this process is repeated for each lobe. In preferredembodiments, the completion of 5 lobes will result in a NavigationAirway Map for the patient.

In various embodiments, the target location for the patient can beselected in the expiration scan and applied to the Navigation AirwayMap. Alternatively, it may be selected in the Inspiration scan andadjusted based on the same or similar criteria as the inspirationairways. In either case, it can be adjusted individually or linked tothe airway via a 3D network and moved in the same transformation.

One aspect of the present invention is directed to a endoscopic portoffset device. The device includes, for example, a snap-on or otherwiseaffixable feature (that is, offsets) to the endoscopic (e.g.,bronchoscopic) port that is capable of holding a navigated guidewire orinstrument in a known or preset (i.e., predetermined) location tomaintain device location and free the physician/user's hand. In oneembodiment, for example, the device includes one or more offset portionsthat can be adjusted by combining and/or removing multiple offsetsegments. In another embodiment, for example, the device includes one ormore offset portions that can be adjusted by the removal of one or moreremovable offset segments separated by perforations (i.e., in adisposable fashion). In yet another embodiment, the device includes anoffset that is capable of adjustment using a screw mechanism (i.e., thelength of the offset can be adjusted by screwing the offset in and out).In various embodiments, each offset can be represented on the navigationscreen showing offset distance from the tip of the endoscope or workingchannel sheath. The endoscope or attachment thereof, in variousembodiments, may include one or more offsets, two or more offsets, threeor more offsets, four or more offsets, or five or more offsets. In otherembodiments, more than five offsets may be included, e.g., 6-12 offsets,6-18 offsets, 6-24 offsets, or more).

Another aspect of the invention is a closed loop system that allows thenavigation system to steer the working channel using shape memoryalloy/metal type materials. An instrument would have tracking sensorslocated at the tip for directional guidance as described herein thatdrive the micro-actuators along the shaft to turn corners. Usingfeedback from multiple sensors along the shaft it is also possible todetermine the maximum points of friction and adjust the shape of thedevice to allow for easier insertion. One simple metric that can beused, for example, is the difference in the shape or bend of the deviceto the segmented pathway of the vessel or airway, described in furtherdetail below.

Other embodiments include, for example, detachable sensors to a fiducialstructure (e.g., a reduced cost patient pad).

In accordance with other embodiments, for example, a sensor as describedherein (e.g., an electromagnetic (EM) sensor) is affixed (preferablypermanently affixed, but may also be removable) to a device orinstrument so that both the device or instrument (or component thereof)and the sensor move together, such that they can be imaged and viewed.In one embodiment, for example, the device is an aspiration needle andthe needle tip and the sensor move together. In another embodiment, forexample, the device is a brush, forceps, or forceps tissue capturemechanism and these components and the sensor move together. In theseand other embodiments, the device may additionally include an actuatinghandle (e.g., finger holds) that is coupled with the sensor, thusallowing movement tracking. These various embodiments advantageouslyallow the device (and components thereof) to be tracked using thesensor, improving overall accuracy and reliability.

In one particular embodiment, a sensor (e.g., an EM sensor) ispositioned at or near the tip of a steerable catheter which furtherincludes a side exiting working channel for forceps, aspiration needle,a brush, combinations thereof, and the like. Using the sensor (which inpreferred embodiments is a 6DOF sensor as described herein), the usercan have the direction of the side exiting working channel defined onthe navigation screen. That is, the image plane that is generated is oneat a side exiting working channel or port, as opposed to a point orposition distal to the device. This will advantageously allow easiertargeting of lesions that may not be directly in the airway, but ratherpartially or even completely outside of the airway. In accordance withan exemplary method of using the device, the catheter is steeredslightly past the target to align the side exiting port/working channel;the sampling component (e.g., forceps, needle, brush) is then extendedout the catheter. The directional aspect of the instrument can be viewedon the navigation screen and a simulated device can be shown todemonstrate to the user the tissue that will be sampled. Theseapplications may be particularly useful in the sampling of lymph nodesthat are outside the patient airways. In some embodiments, for example,the device may be capable of creating an endobronchial ultrasound(EBUS)-like view. For example, an image plane oriented with the workingchannel plane can be created and the instrument can be shown samplingthe target on this plane. In various alternative embodiments, theimage(s) may be oriented in a plane or orthogonally.

Other embodiments include, for example, using EM sensor as LC or energytransmission device. This stored energy could be used to actuate asampling device such as forceps or power a diagnostic sensor.

Other embodiments include, for example, using an elastic tube length ofscope or steerable catheter to add a sensor (e.g., an EM sensor) to adevice. This may take the form of a flexible sheath or tube the lengthof a bronchoscope or steerable catheter that can be added to existingconventional instrumentation. In this way, the conventional device doesnot have to modified. For instance, a very thin wall device can be slidother the length of the scope or catheter and it can be madenavigational.

In accordance with another general aspect, the PTD comprises a pad thatcan be placed on the patient and left for a few hours or a few days. Forexample, such a device could be placed on a patient prior to a 3D imageacquisition such as CT or MR. The device would contain EM (or other)sensors and an EM/LC tank circuit that would allow it be charged up orturned on once localization was ready to commence. The device could bewireless and transmit induced voltage levels back to the system tonavigate in a 3D coordinate space as described herein.

In various embodiments, the device could either have a separate fiducialstructure in a known orientation to the EM sensors, have theconfiguration learned by the system, or have EM sensors that act asfiducials in the 3D image scan.

As discussed herein, auto registration of a device is conducted, atleast initially, by finding the PTD. In general, the PTD needs to bewithin the 3D volume, but for some devices the 3D volume may be toosmall to include both the PTD and the target. To overcome this, a 2D/3Dalgorithm can be employed in accordance with various embodiments,whereby multiple 2D images of a patient are acquired for a 3D volumereconstruction. The complete PTD (which may include one, two or multiple(e.g., at least 3) sensors or fiducials) does not need to be seen ineach image; e.g., some images may only show one sensor or a partthereof, but the entire collection of 2D images can be compiled and usedto find the whole or entire PTD relative to the target. For example, 1802D images (or more or less depending on patient/device position, etc.)can be used to construct a complete 3D volume (where only a fraction(e.g., 10-30) of the 2D images include all or a portion of the PTD. Thisfacilitates easier device placement and allows the user to focus on thetarget, as the PTD and target do not necessarily need to be in the sameinitial 3D volume.

One aspect is directed to recording 3D location and bronchoscopic videoto construct a 3D model of the patient's airway. This 3D video can berecorded over multiple sessions (e.g., weeks between recording) andcolor, size, and shape analysis/change can be determined and/or comparedfor diagnostic purposes. Not only can airway lumen size and/or shape becompared, but a deformation or vector field can also be compared for themultiple sessions. This can be particularly valuable in determiningoverall lung function change as well as local changes to muscle andtissue elasticity.

Another similar aspect is directed to recording 3D location and EBUSvideo or images to construct a 3D model of the patient's airway or alesion. Typically, the EBUS image is very small and 2D. Thus, recordingmultiple planes of EBUS can be used to create a 3D image of the lesion,lymph node, or blood vessels. Providing correlated CT information to theEBUS image can be valuable in determining the location of structures inthe patient.

Yet another similar aspect is directed to recording 3D location and IVUSvideo or images to construct a 3D model of the patient's blood vesselsor plaques. Like EBUS, the IVUS image is typically very small and 2D.Recording multiple planes of IVUS can be used to create a 3D image ofthe blood vessels and malformations. Providing correlated CT informationto the IVUS image can be valuable in determining the location ofstructures in the patient. In accordance with this and other aspects,OCT may additionally or alternatively employed.

In general, the methods described herein involve increasing registrationaccuracy for Ultrasound (US) to CT or any other 3D image dataset such asMR, CT-PET, and 3D Ultrasound. Using 4D tracking of the patientrespiratory signal and collecting a cine loop of ultrasound images, onecan maximize the US to CT fusion accuracy by limiting the point or planeselection for registration to the correct respiratory cycle that matchesthe 3D dataset. The process involves the use of a patient tracker on thepatient and a tracker on the US transducer; using a localizer (such asan EM localization system) to record a cine loop of US images and matchthe images in the cine loop to the respiratory signal.

Collecting a cine loop of US data for registration is valuable in thatthe user does not necessarily have to select a moving point while thepatient is breathing. The user can simply scroll through the cine loopand, with an indicator of the point in the respiratory cycle, select thepoints that are best used for registration from a static image.

The user then selects, for example, at least 3 points in the US & CTdatasets, or a plane and at least one point to register the US space tothe 3D CT dataset space. Preferably, the points and plane are selectedat the same respiratory point as the 3D CT dataset. Normally, forexample, the 3D CT dataset would be acquired at exhalation. Therefore,the user preferably selects registration points at exhalation orsignificant errors can be incorporated into the US to CT registration.If a user was to select a plane that was at a different point in therespiratory cycle, for instance, there would be significant translationerror in the registration. There would likely also be significantrotational error in the registration if the points were acquired atdifferent points in the respiratory cycle.

The methods and systems described herein can be expanded to use otherpatient sensing information such as cardiac information (heartbeat) as asingle source 4D signal or multiple sensed signal approach in thatrespiration and cardiac data could be used together. This isparticularly relevant in connection with locations close to the heart orwithin the heart.

Preferably, the user selects points that are sufficiently far apart forthe best accuracy. This can be done, for example, by requiring the userto record a cine loop of data that extends over the whole patient organ.Preferably, the user is not allowed to pick multiple points in the sameimage.

Another technique for maximizing registration accuracy is a centroidfinding algorithm that can be used for refining point locations in alocal area. Often, a user will want to select a vessel bifurcation. Thevessel bifurcation will be seen as a bright white location on the CT andUS images. An algorithm can be used to help the user select the optimalcenter location for these locations. Once a user selects a point on theimage, the local algorithm can be employed to find similar white voxelsthat are connected and, for that shape in the 3D space, refine the pointto the centroid or any other optimal point (such as, for example, themost anterior or most posterior point).

In general, the systems and methods described herein can be implementedregardless of the number of sensors that are used. In some embodiments,serial orientation or positioning of multiple sensors allows thedetermination of one or more parameters such as shape, position,orientation, and mechanical status of a complete or partial section ofguidewire or other device or instrument. For example, the placement ofmultiple sensors can assist in visualizing the shape of the device andany bends in the path by providing a number of data points on the path(e.g., 8 sensors, spaced 1 mm apart) to create a 3D shape model of thedevice. Various parameters can be used to track past or present movementand changes in device shape including, for example, elasticity, bendradius, limiting, and durometer rating of the device material. Theseparameters and accompanying data can provide visual cues to the userduring the procedure, for example, when the device has a certain bend orcurvature (based on path or surroundings), e.g., to provide a notice orwarning that the device is on the correct or incorrect path, or toprovide notice regarding, or track, a particular parameter(s) that theuser is interested in. Such a sensor pathway is generally depicted inFIG. 18, which shows exemplary curvature warning scenarios in thedifferently marked sections or segments.

In various aspects and embodiments described herein, one can use theknowledge of the path traveled by the instrument and segmented airway orvessel from the acquired image (e.g., CT) to limit the possibilities ofwhere the instrument is located in the patient. The techniques describedherein, therefore, can be valuable to improve virtual displays forusers. Fly through, Fly-above, or image displays related to segmentedpaths are commonly dependent upon relative closeness to the segmentedpath. For a breathing patient, for example, or a patient with a movingvessel related to heartbeat, it is valuable to use the path traveledinformation to determine where in the 4D patient motion cycle the systemis located within the patient. By comparing the 3D location, thepatient's tracked or physiological signal is used to determine 4Dpatient motion cycle, and with the instrument's traveled path, one candetermine the optical location relative to a segmented airway or vesseland use this information to provide the optimal virtual display.

FIGS. 1 and 2 are schematic illustrations of devices that can be used inconjunction with, or to perform, various procedures described herein. Asshown in FIG. 1, an apparatus 10 includes a PTD 20. The PTD 20 can becoupled to a dynamic body B. The dynamic body B can be, for example, aselected dynamic portion of the anatomy of a patient. The PTD 20 can bea variety of different shapes and sizes. For example, in one embodimentthe PTD 20 is substantially planar, such as in the form of a patch thatcan be disposed at a variety of locations on a patient's body. Such aPTD 20 can be coupled to the dynamic body with adhesive, straps, hookand pile, snaps, or any other suitable coupling method. In anotherembodiment the PTD can be a catheter type device with a pigtail oranchoring mechanism that allows it to be attached to an internal organor along a vessel.

Two or more markers or fiducials 22 are coupled to the PTD 20 atselected locations as shown in FIG. 1. The markers 22 are constructed ofa material that can be viewed on an image, such as an X-ray or CT. Themarkers 22 can be, for example, radiopaque, and can be coupled to thePTD 20 using any known methods of coupling such devices. FIGS. 1 and 2illustrate the apparatus 10 having four markers 22, but any number oftwo or more markers can be used. In one embodiment the marker orfiducials and the localization element can be the same device.

An imaging device 40 can be used to take images of the dynamic body Bwhile the PTD 20 is coupled to the dynamic body B, pre-procedurallyduring a first time interval. As stated above, the markers 22 arevisible on the images and can provide an indication of a position ofeach of the markers 22 during the first time interval. The position ofthe markers 22 at given instants in time through a path of motion of thedynamic body B can be illustrated with the images. The imaging device 40can be, for example, a computed tomography (CT) device (e.g.,respiratory-gated CT device, ECG-gated CT device), a magnetic resonanceimaging (MRI) device (e.g., respiratory-gated MRI device, ECG-gated MRIdevice), an X-ray device, or any other suitable medical imaging device.In one embodiment, the imaging device 40 is a computedtomography—positron emission tomography device that produces a fusedcomputed tomography—positron emission tomography image dataset. Theimaging device 40 can be in communication with a processor 30 and send,transfer, copy and/or provide image data taken during the first timeinterval associated with the dynamic body B to the processor 30.

The processor 30 includes a processor-readable medium storing coderepresenting instructions to cause the processor 30 to perform aprocess. The processor 30 can be, for example, a commercially availablepersonal computer, or a less complex computing or processing device thatis dedicated to performing one or more specific tasks. For example, theprocessor 30 can be a terminal dedicated to providing an interactivegraphical user interface (GUI). The processor 30, according to one ormore embodiments of the invention, can be a commercially availablemicroprocessor. Alternatively, the processor 30 can be anapplication-specific integrated circuit (ASIC) or a combination ofASICs, which are designed to achieve one or more specific functions, orenable one or more specific devices or applications. In yet anotherembodiment, the processor 30 can be an analog or digital circuit, or acombination of multiple circuits.

The processor 30 can include a memory component 32. The memory component32 can include one or more types of memory. For example, the memorycomponent 32 can include a read only memory (ROM) component and a randomaccess memory (RAM) component. The memory component can also includeother types of memory that are suitable for storing data in a formretrievable by the processor 30. For example, electronicallyprogrammable read only memory (EPROM), erasable electronicallyprogrammable read only memory (EEPROM), flash memory, as well as othersuitable forms of memory can be included within the memory component.The processor 30 can also include a variety of other components, such asfor example, coprocessors, graphic processors, etc., depending upon thedesired functionality of the code.

The processor 30 can store data in the memory component 32 or retrievedata previously stored in the memory component 32. The components of theprocessor 30 can communicate with devices external to the processor 30by way of an input/output (I/O) component (not shown). According to oneor more embodiments of the invention, the I/O component can include avariety of suitable communication interfaces. For example, the I/Ocomponent can include, for example, wired connections, such as standardserial ports, parallel ports, universal serial bus (USB) ports, S-videoports, local area network (LAN) ports, small computer system interface(SCCI) ports, and so forth. Additionally, the I/O component can include,for example, wireless connections, such as infrared ports, opticalports, Bluetooth® wireless ports, wireless LAN ports, or the like.

The processor 30 can be connected to a network, which may be any form ofinterconnecting network including an intranet, such as a local or widearea network, or an extranet, such as the World Wide Web or theInternet. The network can be physically implemented on a wireless orwired network, on leased or dedicated lines, including a virtual privatenetwork (VPN).

As stated above, the processor 30 can receive image data from theimaging device 40. The processor 30 can identify the position ofselected markers 22 within the image data or voxel space using varioussegmentation techniques, such as Hounsfield unit thresholding,convolution, connected component, or other combinatory image processingand segmentation techniques. The processor 30 can determine a distanceand direction between the position of any two markers 22 during multipleinstants in time during the first time interval, and store the imagedata, as well as the position and distance data, within the memorycomponent 32. Multiple images can be produced providing a visual imageat multiple instants in time through the path of motion of the dynamicbody. The processor 30 can also include a receiving device orlocalization device 34, which is described in more detail below.

A deformation field may also be included in the analysis in variousembodiments described herein. For example, the deformation field can beapplied to fuse 3D fluoroscopic images to CT images in order tocompensate for different patient orientations, patient position,respiration, deformation induced by the catheter or other instrument,and/or other changes or perturbations that occur due to therapy deliveryor resection or ablation of tissue.

In some embodiments, for example, real-time respiration compensation canbe determined by applying an inspiration-to-expiration deformationvector field. In combination with the PTD respiratory signal, forexample, the instrument location can be calculated using the deformationvector field. A real-time instrument tip correction vector can beapplied to a 3D localized instrument tip. The real-time correctionvector is computed by scaling an inspiration-to-expiration deformationvector (found from the inspiration-to-expiration deformation vectorfield) based on the PTD respiratory signal. This correction vector canthen be applied to the 3D localized instrument tip. This can furtheroptimize accuracy during navigation.

An example of an algorithm for real-time respiration compensation can befound in FIG. 19. In accordance with this algorithm, for each l:

(a) find v_(i) such that scalar d is minimized;

(b) compute c, wherein:

c=−v _(i) t

-   -   and (c) compute l′, wherein:

l′=l+c

Thus, l′ is a respiration compensated version of l.

Although FIG. 19 and the above discussion generally relate to real-timerespiration motion, it will be understood that these calculations anddeterminations may also be applied to real-time heartbeat and/or vesselmotion compensation, or any other motion of a dynamic body as describedherein. In one embodiment, for example, the deformation matrix iscalculated based upon inspiration and expiration. In another embodiment,for example, the deformation matrix is calculated based upon heartbeat.In yet another embodiment, for example, the deformation matrix is basedupon vessel motion. In these and other embodiments, it is also possibleto extend these calculations and determinations to develop multipledeformation matrices across multiple patient datasets, by acquiring themultiple datasets over the course of, for example, a single heartbeatcycle or a single respiratory cycle.

Deformation on 2D images can also be calculated based upon therapeuticchange of tissue, changes in Houndsfield units for images, patientmotion compensation during the imaging sequence, therapy monitoring, andtemperature monitoring with fluoroscopic imaging, among other things.One potential issue with conventional therapy delivery, for instance, ismonitoring the therapy for temperature or tissue changes. In accordancewith the methods described herein, this monitoring can be carried outusing intermittent fluoroscopic imaging, where the images arecompensated between acquisition times to show very small changes inimage density, which can represent temperature changes or tissue changesas a result of the therapy and/or navigation.

In general, it may also be preferable to reduce the level of radiationthat patients are exposed to before or during a procedure (orpre-procedural analysis) as described herein. One method of reducingradiation during the acquisition of a 3D fluoroscopic dataset (or otherdataset described herein), for example, is to use a deformation fieldbetween acquired 2D images to reduce the actual number of 2D images thatneed to be acquired to create the 3D dataset. In one particularembodiment, the deformation field is used to calculate the deformationbetween images in the acquisition sequence to produce 2D images betweenthe acquired slices, and these new slices can be used to calculate the3D fluoroscopic dataset. For example, if 180 2D image slices werepreviously required, e.g., an image(s) taken every 2 degrees of a 360degree acquisition sequence, in accordance with some embodiments 90 2Dimages can be acquired over a 360 degree acquisition sequence and thedata from the images that would have ordinarily been acquired betweeneach slice can be calculated and imported into the 3D reconstructionalgorithm. Thus, the radiation is effectively reduced by 50%.

As shown in FIG. 2, two or more localization elements 24 are coupled tothe PTD 20 proximate the locations of the markers 22 for use during amedical procedure to be performed during a second time interval. Thelocalization elements 24 can be, for example, electromagnetic coils,infrared light emitting diodes, and/or optical passive reflectivemarkers. The localization elements 24 can also be, or be integratedwith, one or more fiber optic localization (FDL) devices. The markers 22can include plastic or non-ferrous fixtures or dovetails or othersuitable connectors used to couple the localization elements 24 to themarkers 22. A medical procedure can then be performed with the PTD 20coupled to the dynamic body B at the same location as during the firsttime interval when the pre-procedural images were taken. During themedical procedure, the localization elements 24 are in communication orcoupled to the localization device 34 included within processor 30. Thelocalization device 34 can be, for example, an analog to digitalconverter that measures voltages induced onto localization coils in thefield; creates a digital voltage reading; and maps that voltage readingto a metric positional measurement based on a characterized volume ofvoltages to millimeters from a fixed field emitter. Position dataassociated with the elements 24 can be transmitted or sent to thelocalization device 34 continuously during the medical procedure duringthe second time interval. Thus, the position of the localizationelements 24 can be captured at given instants in time during the secondtime interval. Because the localization elements 24 are coupled to thePTD 20 proximate the markers 22, the localization device 34 can use theposition data of the elements 24 to deduce coordinates or positionsassociated with the markers 22 intra-procedurally during the second timeinterval. The distance, range, acceleration, and speed between one ormore selected pairs of localization elements 24 (and correspondingmarkers 22) can then be determined and various algorithms can be used toanalyze and compare the distance between selected elements 24 at giveninstants in time, to the distances between and orientation amongcorresponding markers 22 observed in the pre-operative images.

An image can then be selected from the pre-operative images taken duringthe first time interval that indicates a distance or is grouped in asimilar sequence of motion between corresponding markers 22 at a giveninstant in time, that most closely approximates or matches the distanceor similar sequence of motion between the selected elements 24. Theprocess of comparing the distances is described in more detail below.Thus, the apparatus 10 and processor 30 can be used to provide imagescorresponding to the actual movement of the targeted anatomy during themedical procedure being performed during the second time interval. Theimages illustrate the orientation and shape of the targeted anatomyduring a path of motion of the anatomy, for example, during inhaling andexhaling.

FIG. 3 illustrates an example set of distances or vectors d1 through d6between a set of markers 122, labeled m1 through m9 that are disposed atspaced locations on a PTD 120. As described above, pre-procedure imagescan be taken of a dynamic body for which the PTD 120 is to be coupledduring a first time interval. The distances between the markers can bedetermined for multiple instants in time through the path of motion ofthe dynamic body. Then, during a medical procedure, performed during asecond time interval, localization elements (not shown in FIG. 3)coupled proximate to the location of markers 122 can provide positiondata for the elements to a localization device (not shown in FIG. 3).The localization device can use the position data to determine distancesor vectors between the elements for multiple instants in time during themedical procedure or second time interval.

FIG. 4A shows an example of distance or vector data from thelocalization device. Vectors a1 through a6 represent distance data forone instant in time and vectors n1 through n6 for another instant intime, during a time interval from a to n. As previously described, thevector data can be used to select an image from the pre-proceduralimages that includes distances between the markers m1 through m9 thatcorrespond to or closely approximate the distances a1 through a6 fortime a, for example, between the localization elements. The same processcan be performed for the vectors n1 through n6 captured during time n.

One method of selecting the appropriate image from the pre-proceduralimages is to execute an algorithm that can sum all of the distances a1through a6 and then search for and match this sum to an image containinga sum of all of the distances d1 through d6 obtained pre-procedurallyfrom the image data that is equal to the sum of the distances a1 througha6. When the difference between these sums is equal to zero, therelative position and orientation of the anatomy or dynamic body Dduring the medical procedure will substantially match the position andorientation of the anatomy in the particular image. The image associatedwith distances d1 through d6 that match or closely approximate thedistances a1 through a6 can then be selected and displayed. For example,FIG. 4B illustrates examples of pre-procedural images, Image a and Imagen, of a dynamic body D that correspond to the distances a1 through a6and n1 through n6, respectively. An example of an algorithm fordetermining a match is as follows:

Does Σa_(i)=Σd_(i) (i=1 to 6 in this example) OR

Does Σ(a_(i)−d_(i))=0 (i=1 to 6 in this example).

If yes to either of these, then the image is a match to the vector ordistance data obtained during the medical procedure.

FIG. 5 illustrates an apparatus 210 according to an embodiment of theinvention. The apparatus 210 includes a tubular shaped PTD 220 that canbe constructed with a rigid material or, alternatively, a flexibleand/or stretchable material. In one embodiment, for example, the PTD 220is substantially rigid in structure. In another embodiment, for example,the PTD 220 has a flexible or stretchable structure. The PTD 220 can bepositioned over a portion of a patient's body, such as around the upperor lower torso of the patient. In the embodiments in which the PTD 220is constructed with a stretchable and/or flexible material, forinstance, the stretchability of the PTD 220 allows the PTD 220 to atleast partially constrict some of the movement of the portion of thebody for which it is coupled. The apparatus 210 further includesmultiple markers or fiducials 222 coupled to the PTD 220 at spacedlocations. A plurality of localization elements 224 are removablycoupled proximate to the locations of markers 222, such that during afirst time interval as described above, images can be taken without theelements 224 being coupled to the PTD 220. The localization elementsneed not be removably coupled. For example, the elements can be fixedlycoupled to the PTD. In addition, the elements can be coupled to the PTDduring the pre-procedure imaging.

FIG. 6 is a graphical illustration indicating how the apparatus 210(shown without localization elements 224) can move and changeorientation and shape during movement of a dynamic body, such as amammalian body M. The graph is one example of how the lung volume canchange during inhalation (inspiration) and exhalation (expiration) ofthe mammalian body M. The corresponding changes in shape and orientationof the apparatus 210 during inhalation and exhalation are alsoillustrated. The six markers 222 shown in FIG. 5 are labeled a, b, c, d,e, and f. As described above, images of the apparatus 110 can be takenduring a first time interval. The images can include an indication ofrelative position of each of the markers 222, that is the markers 222are visible in the images, and the position of each marker 222 can thenbe observed over a period of time. A distance between any two markers222 can then be determined for any given instant of time during thefirst time interval. For example, a distance X between markers a and bis illustrated, and a distance Y between markers b and f is illustrated.These distances can be determined for any given instant in time duringthe first time interval from an associated image that illustrates theposition and orientation of the markers 222. As illustrated, duringexpiration of the mammalian body M at times indicated as A and C, thedistance X is smaller than during inspiration of the mammalian body M,at the time indicated as B. Likewise, the distance Y is greater duringinspiration than during expiration. The distance between any pair ofmarkers 222 can be determined and used in the processes describedherein. Thus, the above embodiments are merely examples of possible pairselections. For example, a distance between a position of marker e and aposition of marker b may be determined. In addition, multiple pairs oronly one pair may be selected for a given procedure.

FIG. 7 is a flowchart illustrating a method according to an embodimentof the invention. A method 50 includes at step 52 receiving image dataduring a pre-procedural or first time interval. As discussed above,images are taken of a dynamic body using an appropriate imaging modality(e.g., CT Scan, MRI, etc.). The image data is associated with one ormore images taken of a PTD (as described herein) coupled to a dynamicbody, where the PTD includes two or more markers coupled thereto. Inother words, the image data of the dynamic body is correlated with imagedata related to the PTD. The one or more images can be taken using avariety of different imaging modalities as described previously. Theimage data can include an indication of a position of a first marker andan indication of a position of a second marker, as illustrated at step54. The image data can include position data for multiple positions ofthe markers during a range or path of motion of the dynamic body over aselected time interval. As described above, the image data can includeposition data associated with multiple markers, however, only two aredescribed here for simplicity. A distance between the position of thefirst marker and the position of the second marker can be determined formultiple instants in time during the first time interval, at step 56. Asalso described above, the determination can include determining thedistance based on the observable distance between the markers on a givenimage. The image data, including all of the images received during thefirst time interval, the position, and the distance data can be storedin a memory and/or recorded at step 58.

Then at step 60, during a second time interval, while performing amedical procedure on the patient with the PTD positioned on the patientat substantially the same location, position data can be received for afirst localization element and a second localization element. Thelocalization elements can be coupled to the PTD proximate the locationsof the markers, such that the position data associated with the elementscan be used to determine the relative position of the markers inreal-time during the medical procedure. The position data of theelements can be stored and/or recorded at step 62.

A distance between the first and second localization elements can bedetermined at step 64. Although only two localization elements aredescribed, as with the markers, position data associated with more thantwo localization elements can be received and the distances between theadditional elements can be determined.

The next step is to determine which image from the one or more imagestaken during the first time interval represents the relative positionand/or orientation of the dynamic body at a given instant in time duringthe second time interval or during the medical procedure. To determinethis, at step 66, the distance between the positions of the first andsecond localization elements at a given instant in time during thesecond time interval are compared to the distance(s) determined in step56 between the positions of the first and second markers obtained withthe image data during the first time interval.

An image can be selected from the first time interval that bestrepresents the same position and orientation of the dynamic body at agiven instant in time during the medical procedure. To do this, thedifference between the distance between a given pair of localizationelements during the second time interval is used to select the imagethat contains the same distance between the same given pair of markersfrom the image data received during the first time interval. This can beaccomplished, for example, by executing an algorithm to perform thecalculations. When there are multiple pairs of markers and localizationelements, the algorithm can sum the distances between all of theselected pairs of elements for a given instant in time during the secondtime interval and sum the distances between all of the associatedselected pairs of markers for each instant in time during the first timeinterval when the pre-procedural image data was received.

When an image is found that provides the sum of distances for theselected pairs of markers that is substantially the same as the sum ofthe distances between the localization elements during the second timeinterval, then that image is selected at step 68. The selected image canthen be displayed at step 70. The physician can then observe the imageduring the medical procedure on a targeted portion of the dynamic body.Thus, during the medical procedure, the above process can becontinuously executed such that multiple images are displayed and imagescorresponding to real-time positions of the dynamic body can be viewed.

FIG. 8 shows one embodiment of a system (system 100) that includescomponents that can be used to perform image guided interventions usinga gated imaging modality, such as ECG-gated MRI, or ECG-gated CT. Thefigure depicts a patient 10 positioned on an operating table 12 with aphysician 14 performing a medical procedure on him.

Specifically, FIG. 8 depicts physician 14 steering a medical instrument16 through the patient's internal anatomy in order to deliver therapy.In this particular instance, instrument 16 is depicted as a catheterentering the right atrium by way of the inferior vena cava preceded by afemoral artery access point; however, the present systems are notlimited to catheter use indications. The position of virtually anyinstrument may be tracked as discussed below and a representation of itsuperimposed on the proper image, consistent with the present methods,apparatuses, and systems. An “instrument” is any device controlled byphysician 10 for the purpose of delivering therapy, and includesneedles, guidewires, stents, filters, occluders, retrieval devices,imaging devices (such as OCT, EBUS, IVUS, and the like), and leads.Instrument 16 is fitted with one or more instrument reference markers18. A tracker 20 (which is sometimes referred to in the art as a“tracking system”) is configured to track the type of reference markeror markers coupled to instrument 16. Tracker 20 can be any type oftracking system, including but not limited to an electromagnetictracking system. An example of a suitable electromagnetic trackingsystem is the AURORA electromagnetic tracking system, commerciallyavailable from Northern Digital Inc. in Waterloo, Ontario Canada. Iftracker 20 is an electromagnetic tracking system, element 20 wouldrepresent an electromagnetic field generator that emits a series ofelectromagnetic fields designed to engulf patient 10, and referencemarker or markers 18 coupled to medical instrument 16 could be coilsthat would receive an induced voltage that could be monitored andtranslated into a coordinate position of the marker(s).

As noted herein, a variety of instruments and devices can be used inconjunction with the systems and methods described herein. In oneembodiment, for example, an angled coil sensor is employed during thetargeted navigation. In accordance with this embodiment, for example,instead of using a conventional wire sensor wrapped at about a 90° angle(i.e., roughly perpendicular) to the axial length (or core) of thesensor, the coil is wrapped at an acute angle (i.e., the angle is lessthan about 90°) relative to the axial length of the sensor. In oneembodiment, the coil is positioned (e.g., wrapped) at an angle of fromabout 30° to about 60° relative to the axial length. In one preferredembodiment, the coil is positioned at about a 45° angle relative to theaxial length. The positioning of the coil in accordance with theexemplary embodiments described herein advantageously provides adirectional vector that is not parallel with the sensor core. Thus, thephysical axis is different and, as the sensor moves, this additionaldirectional vector can be quantified and used to detect up and down (andother directional) movement. This motion can be captured over time asdescribed herein to determine orientation and prepare and display themore accurate images.

An external reference marker 22 can be placed in a location close to theregion of the patient where the procedure is to be performed, yet in astable location that will not move (or that will move a negligibleamount) with the patient's heart beat and respiration. If patient 10 issecurely fixed to table 12 for the procedure, external reference marker22 (which may be described as “static”) can be affixed to table 12. Ifpatient 10 is not completely secured to table 12, external referencemarker 22 can be placed on region of the back of patient 10 exhibitingthe least amount of movement. Tracker 20 can be configured to trackexternal reference marker 22.

One or more non-tissue internal reference markers 24 can be placed inthe gross region where the image guided navigation will be carried out.Non-tissue internal reference marker(s) 24 should be placed in ananatomic location that exhibits movement that is correlated with themovement of the anatomy intended for image guided navigation. Thislocation will be internal to the patient, in the gross location of theanatomy of interest.

Medical instrument 16, instrument reference marker(s) 18, externalreference marker 22, and non-tissue internal reference marker(s) 24 canbe coupled to converter 26 of system 100. Converter 26, one example ofwhich may be referred to in the art as a break-out box, can beconfigured to convert analog measurements received from the referencemarkers and tracker 20 into digital data understandable by imageguidance computing platform 30, and relay that data to image guidancecomputing platform 30 to which converter 26 can be coupled. Imageguidance computing platform 30 can take the form of a computer, and mayinclude a monitor on which a representation of one or more instrumentsused during the IGI can be displayed over an image of the anatomy ofinterest.

System 100 also includes a periodic human characteristic signal monitor,such as ECG monitor 32, which can be configured to receive a periodichuman characteristic signal. For example, ECG monitor 32 can beconfigured to receive an ECG signal in the form of the ECG datatransmitted to it by ECG leads 34 coupled to patient 10. The periodichuman characteristic signal monitor (e.g., ECG monitor 32) can also beconfigured to relay a periodic human characteristic signal (e.g., ECGdata) to image guidance computing platform 30, to which it can becoupled.

Prior to the start of the image guided intervention, non-tissue internalreference marker(s) 24—but not necessarily static external referencemarker 22—should be placed in the gross region of interest for theprocedure. After placement of non-tissue internal reference marker(s)24, patient 10 is to be scanned with an imaging device, such as gatedscanner 40, and the resulting gated image dataset transferred to imageguidance computing platform 30, to which the imaging device is coupledand which can reside in the operating or procedure theatre. Examples ofsuitable imaging devices, and more specifically suitable gated scanners,include ECG-gated MRI scanners and ECG-gated CT scanners. A hospitalnetwork 50 may be used to couple gated scanner 40 to image guidancecomputing platform 30.

The imaging device (e.g., gated scanner 40) can be configured to createa gated dataset that includes pre-operative images, one or more of which(up to all) are taken using the imaging device and are linked to asample of a periodic human characteristic signal (e.g., a sample, or aphase, of an ECG signal). Once patient 10 is scanned using the imagingdevice and the gated dataset is transferred to and received by imageguidance computing platform 30, patient 10 can be secured to operatingtable 12 and the equipment making up system 100 (e.g., tracker 20,converter 26, image guidance computing platform 30, ECG monitor 32, andgated scanner 40) set up as shown in FIG. 9. Information can then flowamong the system 100 components.

At this point, a gated dataset created by gated scanner 40 resides onimage guidance computing platform 30. FIG. 9 highlights the relationshipbetween the samples (S1 . . . Sn) and the images (I1 . . . In) that werecaptured by gated scanner 40. Designations P, Q, R, S, and T aredesignations well known in the art; they designate depolarizations andre-polarizations of the heart. Gated scanner 40 essentially creates animage of the anatomy of interest at a particular instant in time duringthe anatomy's periodic movement. Image I1 corresponds to the image thatwas captured at the S1 moment of patient 10's ECG cycle. Similarly, I2is correlated with S2, and In with Sn.

FIG. 10 is a diagram of another exemplary surgical instrument navigationsystem 10. In accordance with one aspect of the present invention, thesurgical instrument navigation system 10 is operable to visuallysimulate a virtual volumetric scene within the body of a patient, suchas an internal body cavity, from a point of view of a surgicalinstrument 12 residing in the cavity of a patient 13. To do so, thesurgical instrument navigation system 10 is primarily comprised of asurgical instrument 12, a data processor 16 having a display 18, and atracking subsystem 20. The surgical instrument navigation system 10 mayfurther include (or accompanied by) an imaging device 14 that isoperable to provide image data to the system.

The surgical instrument 12 is preferably a relatively inexpensive,flexible and/or steerable catheter that may be of a disposable type. Thesurgical instrument 12 is modified to include one or more trackingsensors that are detectable by the tracking subsystem 20. It is readilyunderstood that other types of surgical instruments (e.g., a guide wire,a needle, a forcep, a pointer probe, a stent, a seed, an implant, anendoscope, an energy delivery device, a therapy delivery device, etc.)are also within the scope of the present invention. It is alsoenvisioned that at least some of these surgical instruments may bewireless or have wireless communications links. It is also envisionedthat the surgical instruments may encompass medical devices which areused for exploratory purposes, testing purposes or other types ofmedical procedures.

The volumetric scan data is then registered as shown at 34. Registrationof the dynamic reference frame 19 generally relates information in thevolumetric scan data to the region of interest associated with thepatient. This process is referred to as registering image space topatient space. Often, the image space must also be registered to anotherimage space. Registration is accomplished through knowledge of thecoordinate vectors of at least three non-collinear points in the imagespace and the patient space.

Referring to FIG. 11, the imaging device 14 is used to capturevolumetric scan data 32 representative of an internal region of interestwithin the patient 13. The three-dimensional scan data is preferablyobtained prior to surgery on the patient 13. In this case, the capturedvolumetric scan data may be stored in a data store associated with thedata processor 16 for subsequent processing. However, one skilled in theart will readily recognize that the principles of the present inventionmay also extend to scan data acquired during surgery. It is readilyunderstood that volumetric scan data may be acquired using various knownmedical imaging devices 14, including but not limited to a magneticresonance imaging (MRI) device, a computed tomography (CT) imagingdevice, a positron emission tomography (PET) imaging device, a 2D or 3Dfluoroscopic imaging device, and 2D, 3D or 4D ultrasound imagingdevices. In the case of a two-dimensional ultrasound imaging device orother two-dimensional image acquisition device, a series oftwo-dimensional data sets may be acquired and then assembled intovolumetric data as is well known in the art using a two-dimensional tothree-dimensional conversion.

The multi-dimensional imaging modalities described herein may also becoupled with digitally reconstructed radiography (DRR) techniques. Inaccordance with a fluoroscopic image acquisition, for example, radiationpasses through a physical media to create a projection image on aradiation-sensitive film or an electronic image intensifier. Given a 3Dor 4D dataset as described herein, for example, a simulated image can begenerated in conjunction with DRR methodologies. DRR is generally knownin the art, and is described, for example, by Lemieux et al. (Med. Phys.21(11), November 1994, pp. 1749-60).

When a DRR image is created, a fluoroscopic image is formed bycomputationally projecting volume elements, or voxels, of the 3D or 4Ddataset onto one or more selected image planes. Using a 3D or 4D datasetof a given patient as described herein, for example, it is possible togenerate a DRR image that is similar in appearance to a correspondingpatient image. This similarity can be due, at least in part, to similarintrinsic imaging parameters (e.g., projective transformations,distortion corrections, etc.) and extrinsic imaging parameters (e.g.,orientation, view direction, etc.). The intrinsic imaging parameters canbe derived, for instance, from the calibration of the equipment.Advantageously, this provides another method to see the up-and-down (andother directional) movement of the instrument. This arrangement furtherprovides the ability to see how the device moves in an image(s), whichtranslates to improved movement of the device in a patient. An exemplarypathway in accordance with the disclosure herein can be seen in FIG. 17.

A dynamic reference frame 19 is attached to the patient proximate to theregion of interest within the patient 13. To the extent that the regionof interest is a vessel or a cavity within the patient, it is readilyunderstood that the dynamic reference frame 19 may be placed within thepatient 13. To determine its location, the dynamic reference frame 19 isalso modified to include tracking sensors detectable by the trackingsubsystem 20. The tracking subsystem 20 is operable to determineposition data for the dynamic reference frame 19 as further describedbelow.

The volumetric scan data is then registered as shown at 34. Registrationof the dynamic reference frame 19 generally relates information in thevolumetric scan data to the region of interest associated with thepatient. This process is referred to as registering image space topatient space. Often, the image space must also be registered to anotherimage space. Registration is accomplished through knowledge of thecoordinate vectors of at least three non-collinear points in the imagespace and the patient space.

Registration for image guided surgery can be completed by differentknown techniques. First, point-to-point registration is accomplished byidentifying points in an image space and then touching the same pointsin patient space. These points are generally anatomical landmarks thatare easily identifiable on the patient. Second, surface registrationinvolves the user's generation of a surface in patient space by eitherselecting multiple points or scanning, and then accepting the best fitto that surface in image space by iteratively calculating with the dataprocessor until a surface match is identified. Third, repeat fixationdevices entail the user repeatedly removing and replacing a device(i.e., dynamic reference frame, etc.) in known relation to the patientor image fiducials of the patient. Fourth, automatic registration byfirst attaching the dynamic reference frame to the patient prior toacquiring image data. It is envisioned that other known registrationprocedures are also within the scope of the present invention, such asthat disclosed in U.S. Ser. No. 09/274,972, filed on Mar. 23, 1999,entitled “NAVIGATIONAL GUIDANCE VIA COMPUTER-ASSISTED FLUOROSCOPICIMAGING”, which is hereby incorporated by reference.

FIG. 12 illustrates another type of secondary image 28 which may bedisplayed in conjunction with the primary perspective image 38. In thisinstance, the primary perspective image is an interior view of an airpassage within the patient 13. The secondary image 28 is an exteriorview of the air passage which includes an indicia or graphicalrepresentation 29 that corresponds to the location of the surgicalinstrument 12 within the air passage. In FIG. 12, the indicia 29 isshown as a crosshairs. It is envisioned that other indicia may be usedto signify the location of the surgical instrument in the secondaryimage. As further described below, the secondary image 28 is constructedby superimposing the indicia 29 of the surgical instrument 12 onto themanipulated image data 38.

Referring to FIG. 13, the display of an indicia of the surgicalinstrument 12 on the secondary image may be synchronized with ananatomical function, such as the cardiac or respiratory cycle, of thepatient. In certain instances, the cardiac or respiratory cycle of thepatient may cause the surgical instrument 12 to flutter or jitter withinthe patient. For instance, a surgical instrument 12 positioned in ornear a chamber of the heart will move in relation to the patient's heartbeat. In these instance, the indicia of the surgical instrument 12 willlikewise flutter or jitter on the displayed image 40. It is envisionedthat other anatomical functions which may effect the position of thesurgical instrument 12 within the patient are also within the scope ofthe present invention.

To eliminate the flutter of the indicia on the displayed image 40,position data for the surgical instrument 12 is acquired at a repetitivepoint within each cycle of either the cardiac cycle or the respiratorycycle of the patient. As described above, the imaging device 14 is usedto capture volumetric scan data 42 representative of an internal regionof interest within a given patient. A secondary image may then berendered 44 from the volumetric scan data by the data processor 16.

In order to synchronize the acquisition of position data for thesurgical instrument 12, the surgical instrument navigation system 10 mayfurther include a timing signal generator 26. The timing signalgenerator 26 is operable to generate and transmit a timing signal 46that correlates to at least one of (or both) the cardiac cycle or therespiratory cycle of the patient 13. For a patient having a consistentrhythmic cycle, the timing signal might be in the form of a periodicclock signal. Alternatively, the timing signal may be derived from anelectrocardiogram signal from the patient 13. One skilled in the artwill readily recognize other techniques for deriving a timing signalthat correlate to at least one of the cardiac or respiratory cycle orother anatomical cycle of the patient.

As described above, the indicia of the surgical instrument 12 tracks themovement of the surgical instrument 12 as it is moved by the surgeonwithin the patient 13. Rather than display the indicia of the surgicalinstrument 12 on a real-time basis, the display of the indicia of thesurgical instrument 12 is periodically updated 48 based on the timingsignal from the timing signal generator 26. In one exemplary embodiment,the timing generator 26 is electrically connected to the trackingsubsystem 20. The tracking subsystem 20 is in turn operable to reportposition data for the surgical instrument 12 in response to a timingsignal received from the timing signal generator 26. The position of theindicia of the surgical instrument 12 is then updated 50 on the displayof the image data. It is readily understood that other techniques forsynchronizing the display of an indicia of the surgical instrument 12based on the timing signal are within the scope of the presentinvention, thereby eliminating any flutter or jitter which may appear onthe displayed image 52. It is also envisioned that a path (or projectedpath) of the surgical instrument 12 may also be illustrated on thedisplayed image data 52.

In another aspect of the present invention, the surgical instrumentnavigation system 10 may be further adapted to display four-dimensionalimage data for a region of interest as shown in FIG. 14. In this case,the imaging device 14 is operable to capture volumetric scan data 62 foran internal region of interest over a period of time, such that theregion of interest includes motion that is caused by either the cardiaccycle or the respiratory cycle of the patient 13. A volumetricperspective view of the region may be rendered 64 from the volumetricscan data 62 by the data processor 16 as described above. Thefour-dimensional image data may be further supplemented with otherpatient data, such as temperature or blood pressure, using coloringcoding techniques.

In order to synchronize the display of the volumetric perspective viewin real-time with the cardiac or respiratory cycle of the patient, thedata processor 16 is adapted to receive a timing signal from the timingsignal generator 26. As described above, the timing signal generator 26is operable to generate and transmit a timing signal that correlates toeither the cardiac cycle or the respiratory cycle of the patient 13. Inthis way, the volumetric perspective image may be synchronized 66 withthe cardiac or respiratory cycle of the patient 13. The synchronizedimage 66 is then displayed 68 on the display 18 of the system. Thefour-dimensional synchronized image may be either (or both of) theprimary image rendered from the point of view of the surgical instrumentor the secondary image depicting the indicia of the position of thesurgical instrument 12 within the patient 13. It is readily understoodthat the synchronization process is also applicable to two-dimensionalimage data acquire over time.

To enhance visualization and refine accuracy of the displayed imagedata, the surgical navigation system can use prior knowledge such as thesegmented vessel or airway structure to compensate for error in thetracking subsystem or for inaccuracies caused by an anatomical shiftoccurring since acquisition of scan data. For instance, it is known thatthe surgical instrument 12 being localized is located within a givenvessel or airway and, therefore should be displayed within the vessel orairway. Statistical methods can be used to determine the most likelylocation; within the vessel or airway with respect to the reportedlocation and then compensate so the display accurately represents theinstrument 12 within the center of the vessel or airway. The center ofthe vessel or airway can be found by segmenting the vessels or airwaysfrom the three-dimensional datasets and using commonly known imagingtechniques to define the centerline of the vessel or airway tree.Statistical methods may also be used to determine if the surgicalinstrument 12 has potentially punctured the vessel or airway. This canbe done by determining the reported location is too far from thecenterline or the trajectory of the path traveled is greater than acertain angle (worse case 90 degrees) with respect to the vessel orairway. Reporting this type of trajectory (error) is very important tothe clinicians. The tracking along the center of the vessel may also befurther refined by correcting for motion of the respiratory or cardiaccycle, as described above. While navigating along the vessel or airwaytree prior knowledge about the last known location can be used to aid indetermining the new location. The instrument or navigated device mustfollow a pre-defined vessel or airway tree and therefore can not jumpfrom one branch to the other without traveling along a path that wouldbe allowed. The orientation of the instrument or navigated device canalso be used to select the most likely pathway that is being traversed.The orientation information can be used to increase the probability orweight for selected location or to exclude potential pathways andtherefore enhance system accuracy.

The surgical instrument navigation system of the present invention mayalso incorporate atlas maps. It is envisioned that three-dimensional orfour-dimensional atlas maps may be registered with patient specific scandata or generic anatomical models. Atlas maps may contain kinematicinformation (e.g., heart and lung models) that can be synchronized withfour-dimensional image data, thereby supplementing the real-timeinformation. In addition, the kinematic information may be combined withlocalization information from several instruments to provide a completefour-dimensional model of organ motion. The atlas maps may also be usedto localize bones or soft tissue which can assist in determiningplacement and location of implants.

FIG. 20, for example, illustrates in one exemplary embodiment the use ofInspiration/arms-up CT and Expiration/arms-down CT for image guidednavigation purposes. Stage (A) shows an Inspiration/arms-up pathwayregistration; this is, generally speaking, the preferred CT scanacquisition state for automatic segmentation of the tracheo-bronchialtree. Stage (B), on the other hand, shows FRC/arms-down segmentation(wherein FRC refers to the Functional Residual Capacity (i.e., the lungvolume at the end of a normal expiration, when the muscles ofrespiration are completely relaxed; at FRC (and typically at FRC only),the tendency of the lungs to collapse is exactly balanced by thetendency of the chest wall to expand). FRC/arms-down is, generally, thepreferred navigational state for image guided pulmonary navigation.Unfortunately, however, using the inspiration (arms-up) state of thelungs for navigation can contribute significant error to the imageguided navigation. Thus, in accordance with various embodimentsdescribed herein, the airways are segmented using fullinspiration/arms-up CT acquisition, and mapped to a less-fullysegmented, FRC/arms-down CT acquisition. As shown, the results can thenbe conformed to produce a fully segmented, tracheo-bronchial tree, atthe proper spatial representation for patient FRC (Stage (C)).

FIG. 21 depicts a CT minP/maxP volume reading including a scenariowhereby precise navigation (e.g., using an EM sensor) near a targetlesion is carried out with incomplete segmentation results. Thisscenario provides the user with a view or image 1600 using minP (minimumintensity projection or maxP (maximum intensity projection) volumerenderings to simultaneously integrate one or more of: (i) the lastknown segmented airway 1601; (ii) the target(s) 1602; (iii) a visuallydistinct representation of previously traversed paths that are “bad”(i.e., incorrect) 1603; (iv) the distance and angle 1604 to the target(e.g., a target lesion) using a vector fit to the last 1 cm (or so) oftravel (in addition to, or in place of, instantaneous orientationprovided by a 5DOF tip sensor as described herein); and (v) toincorporate user provided “way points” to create a final-approach tube1605 to the target. As described herein, the image 1600 may also providea real-time or simulated real-time rendering of the device 1606 (e.g.,the device tip as shown with a virtual extension 1607 to the target).

Referring now to FIGS. 22A and 22B, one exemplary embodiment of a 4Dthoracic registration is depicted. In general, the 4D dataset may beacquired through respiratory gating or tracheal temporal registration.In accordance with the methods described herein, for example,acceleration of N data collectors (e.g., magnetic or MEMSaccelerometers, for instance) are measured to register the thorax intime and space, using the general formula: dataT_(thorax)=F(t). As shownin FIGS. 22A and 22B, the various sensors 1701 and tracheal sensor 1702provide data as described herein, as does sternum sensor 1703 (e.g., x,y, and z dynamic tracking). Device 1704 (e.g., biopsy needle or otherdevice or instrument described herein) is further capable of trackingposition and trajectory (as described herein).

An exemplary apparatus and method for respiratory 4D data acquisitionand navigation is depicted in FIG. 23. As shown in the upper box, arespiratory problem or issue is scanned (e.g., by a CT and/or MRscanner) and signal S from the PTD is provide to the CT/IR unit (lowerbox). The 4D registration based on the motion of the fiducial andtracker units (which could be, e.g., EM sensors, MEMS devices,combinations thereof, and the like) is provided to the user (shown inFIG. 18 as an interventional radiologist (IR)) on computer C. The systemis capable of displaying the current position of the device tip in theimage data shown to the IR, using the fiducial or tracker locations inthe scan coupled with the real-time motion information provided by thedevice tip (e.g., which can include a sensor as described herein), thusproviding registration.

A representative offset device in accordance in certain embodiments ofthe disclosure herein is depicted in FIGS. 24A and 24B, which show ascope or other instrument 1500 including one or more offset devices 1501at port 1502. The offset device(s) is/are capable of holding the trackedguidewire 1503 (including brush 1504 at the scope tip) in place andprovide a substantially fixed distance or length of extension out of thescope (or virtual scope) to take a sample. It will be understood thatbrush 1504 at the guidewire tip may be any of a variety of devices,including forceps, needles (e.g., a biopsy needle), and the like. Asshown, multiple offsets 1501 can be provided in stages that allowextension of the guidewire, e.g., at 1 cm, 2 cm, 3 cm, and so on,whereby the user can adjust the offsets by removal or repositioning(removed offsets 1501R, for example, are depicted in dashed lines withinthe port in FIG. 15B). Thus, FIG. 24A shows the brush tip 1504 at thetip of the scope (i.e., prior to extension), whereas FIG. 24B shows theguidewire and tip extended (e.g., 2 cm of extension) by the removal orrepositioning of two offsets 1501R.

In FIG. 25 a representative actuatable sensor/forceps device inaccordance with the embodiments described herein is depicted. As shown,the coil of an EM (or other) sensor 1601 disposed in a catheter (orsimilar device) body 1600 can act as a solenoid to actuate a forceps1602. In accordance with the illustrated embodiment, for example, thesolenoid coil is used as an EM sensor in a “passive” mode, but can beactivated by energy stored in an ultra-capacitor 1603 which actuates theforceps via armature 1604 in an “active” mode.

CONCLUSION

While various embodiments of the invention have been described above, itshould be understood that they have been presented by way of exampleonly, and not limitation. Thus, the breadth and scope of the inventionshould not be limited by any of the above-described embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

The previous description of the embodiments is provided to enable anyperson skilled in the art to make or use the invention. While theinvention has been particularly shown and described with reference toembodiments thereof, it will be understood by those skilled in art thatvarious changes in form and details may be made therein withoutdeparting from the spirit and scope of the invention. For example, thePTD, markers and localization elements can be constructed from anysuitable material, and can be a variety of different shapes and sizes,not necessarily specifically illustrated, while still remaining withinthe scope of the invention.

1. An image-guided method comprising directing an endoscope including atip sensor to an anatomical position in a patient; orienting anendoscopic view to a visual orientation of a user; activating a steeringmechanism of the endoscope to agitate the tip sensor in a plane; anddetermining the orientation and location of the tip sensor in thepatient and correlating the tip sensor location and orientation with oneor more patient images.
 2. The image-guided method of claim 1 whereinthe anatomical position is a branching point of a bronchial tree.
 3. Theimage-guided method of claim 1, wherein the tip sensor is an angled coilsensor, the coil being disposed at a 45° angle relative to a coreportion of the tip sensor.
 4. The image-guided method of claim 1,wherein orientation and location determination comprise applying adeformation vector field.
 5. The image-guided method of claim 1, whereinthe one or more patient images are derived from an imaging device inconjunction with a patient tracking device disposed at an externalregion of the patient.
 6. A method of using a guidewire or othernavigated instrument with one to one rotation to continuously align avirtual display view to be consistent with an actual bronchoscopic videoview.
 7. A method of using a video input of the bronchoscope to adjustthe virtual fly-through view to be consistent with a user's normalperspective.
 8. The method of claim 7 wherein video processing andmatching techniques can be used between the real-time video and thevirtual image to align.
 9. A method of using bronchoscopic video toprovide angular information at a current location to provide targetingor directional cues to the user.
 10. The method of claim 9 wherein theangular information is derived from the location of patient anatomy inthe image and the relative size of each within the image.
 11. A systemor apparatus comprising one or more components for carrying out one ormore elements of the method of claim
 7. 12. A non-transitoryprocessor-readable medium storing code representing instructions tocause a processor to perform a process, the code comprising code tocarry out one or more elements of the method of claim
 7. 13. An imagingmethod comprising tracking the traveled path of an instrument in apatient and correlating the tracked path with one or more sets of imagedata derived from an imaging device in conjunction with a patienttracking device.
 14. The method of claim 13 wherein a respiratory signalor a heartbeat signal derived from the patient tracking device is usedto gate localization data of the instrument in an anatomical position ofthe patient to determine on or more patient airway models and correlatethe instrument position to the image data to provide a registration ofthe one or more patient airway models during a respiratory cycle of thepatient.
 15. The method of claim 13 wherein the patient airway modeldetermination comprises the addition of patient airway data to the imagedata.
 16. The method of claim 15 wherein the patient airway datacomprises branching points or segments of branching points in abronchial tree.
 17. The method of claim 13, wherein the correlation ofthe tracked path comprises applying a deformation vector field.
 18. Asystem or apparatus comprising one or more components for carrying outone or more elements of the method of claim
 13. 19. A non-transitoryprocessor-readable medium storing code representing instructions tocause a processor to perform a process, the code comprising code tocarry out one or more elements of the method of claim
 13. 20. Aendoscope or attachment thereof including one or more offset devicesproximate a port of the endoscope or attachment, the offset devicescapable of securing a navigated guidewire in a predetermined location.21. The endoscope or attachment of claim 20 wherein a guidewire of theendoscope comprises one or more detachable sensors on a fiducialstructure.
 22. The endoscope or attachment of claim 20 wherein theendoscope comprises an sensor affixed proximate to an aspiration needledisposed at an end of the endoscope or attachment, wherein the needletip and the sensor move in conjunction with one another upon actuationof the endoscope by a user.
 23. The endoscope or attachment of claim 20wherein the endoscope comprises an sensor affixed proximate to a brushdisposed at an end of the endoscope or attachment, wherein the brush andthe sensor move in conjunction with one another upon actuation of theendoscope by a user.
 24. The endoscope or attachment of claim 20 whereinthe endoscope comprises an sensor affixed proximate to a forcepsdisposed at an end of the endoscope or attachment, wherein the forcepsand the sensor move in conjunction with one another upon actuation ofthe endoscope by a user.
 25. The endoscope or attachment of claim 20,wherein the endoscope or attachment comprises three or more offsetdevices.
 26. The endoscope or attachment of claim 20 wherein theendoscope or attachment includes two or more offsets in a series, eachoffset being positioned 1 cm from each other.
 27. A system or methodcomprising the use of an endoscope or attachment of claim 20 in amedical procedure.
 28. A method of constructing a three-dimensionalmodel of an airway or vessel of a patient comprising correlating athree-dimensional location of instrument at an internal position of thepatient with image data collected in conjunction with a patient trackingdevice disposed on an external position of the patient.
 29. The methodof claim 28 wherein the construction comprises recordingthree-dimensional location of an instrument and corresponding EBUS videoor images to construct a three-dimensional model of the patient's airwayor a lesion therein.
 30. The method of claim 28 wherein the constructioncomprises recording three-dimensional location and corresponding IVUSvideo or images to construct a three-dimensional model of the patient'svessel or a plaque therein.
 31. The method of claim 28 wherein theconstruction comprises recording three-dimensional location andcorresponding OCT images to construct a three-dimensional model of thepatient's vessel or a plaque therein.
 32. The method of claim 28 whereinthe construction comprises the use of a fiber optic localization (FDL)device disposed on an instrument positioned within an airway or vesselof the patient and further comprises generating localization informationand shape sensing information and applying an algorithm to saidlocalization and shape sensing information to determine the location andorientation of the instrument within the patient.
 33. A system orapparatus comprising one or more components for carrying out one or moreelements of the method of claim
 28. 34. A non-transitoryprocessor-readable medium storing code representing instructions tocause a processor to perform a process, the code comprising code tocarry out one or more elements of the method of claim 28.